DOCUMENT RESUME 



ED 04E 347 



Tfl 000 421 



AUTHOR 

TITLE 

PUB DATE 
NOTE 



EDRS FRICF 
DESCRIPTORS 



IDENTIFIERS 



Waldron, James S, 

The Delphi Process: Some Assumptions and Some 
Rea lit ies. 

Feb 7 1 

18p.; Paper presented at the Annual Meeting of the 
American Educational Research Association, New York, 
New York, February 4-7, 1971 

EDRS Price Mf-$0*65 HC-S3.29 

♦Complexity Level, Conceptual Schemes, ♦Decision 
Making, ^Evaluative Thinking, ♦Feedback, Group 
Behavior, Individual Differences, ♦Information 
Processing, Interaction Process Analysis, Models, 
Policy Formation, Prediction, Task Performance, 
Thought Processes 
♦Delphi Technique 



ABSTRACT 

The ef feet iveness of the Delphi Technique is 
evaluated in terms of immediate and delayed controlled information 
fet Ifcack (feedtack within 5 seconds as compared with a 2u-hour 
delay); and t \ e relationships that exist among measures of 
integrative complexity, estimations about the time of occurrence of 
future events, and time delay between task completion i nd the 
reception of controlled feedback. Experimental procedures, 
hypotheses, and findings are discussed as they relate to these 
variabl s. Three tnajoL conclusions drawn from the results of the 
study include: (1) individual integrative complexity is strongly 

predictive of performance on the tasks of raking time estimates of 
occurrence of remote future events, and changing those estimates in 
light of controlled information feedback; (2) individual integrative 
complexity is consistently predictive of convergence; and (3) 
differential time delay of information feedback has a significant 
effe t on the performance of individuals involved in the Delphi 
process. The assumption that the phenomenon of corverg^nce in the 
direction of a "correct solution" is a product of objectivity 
relatively free frem the interference of certain psychological 
variables is considered unfounded. It is also contended that 
non*o b jec t iv e factors are as reflective of outcomes in a Delphi 
exercise as t. w ey are in direct confrontation group outcomes. The 
Interrelatedness of time delay variables and performance in the 
Delphi process is considered. A bioliography is included. (AS) 
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Education, as one of the most important institutions of 
our society, is caught up in this modern world of rapid change 
and uncertain future. Educational policy makers are being 
forced to make policy decisions which will have far-reaching 
impact on the future, and which nay remove the freedom of 
choice from future individuals. These policy decisions are 
being made in the absence of knowledge of their impact and 
are, at best, most often based upon intuitive estimations of 
what the future will be like; at worst, these decisions are 
being made based upon practical experience which may have 
little relevance to tomorrow's needs. But no matter what 
method is used, these decisions are being made ana they do 
and will have far-reaching implications for tomorrow's 
generations. 

Educational policy makers must become aware of as many 
alternatives and their potential outcomes as is possible 
when making decisions. In most cases the policy maker cannot 
be an expert in all areas that might possibly relate to a 
particular question. Usually his decisions are based on the 
advice and alternatives generated by experts from all the 
areas that are directly and indirectly related. In other 
words, these policy decisions are based on human judgments 
and estimations of the future. 

One of the procedures very frequently used for the 
collection of opinions is the Delphi technique.* The Delphi 
technique 

is a carefully designed program of 
sequential individual interrogations 
(usually best conducted by question- 
naire) interspersed with information 
and opinion feedback (Helmer, 1967 , p.7^ . 

The assumed value of this technique lies in its provision of: 
(a) anonymity, (b) iteration with controlled feedback, and 
(c) statistical summarization of responses. Anonymity is 
achieved by using questionnaires where specific responses are 
not associated with individual members of the group. This is 
assumed to lessen the effects of dominant individuals and to 
reduce group pressure. Iteration with controlled feedback 
occurs in several stages; generally, at the beginning of each 
questionnaire round, the results of the previous round are 
statistically summarized and reported back to the participat- 
ing individuals. And, finally, the statistical summarization 
of responses allows a consensus to be reached without asking 
the group to arrive at a common opinion. It is assumed that 



*The Delphi technique was developed by Helmer (I960) to 
systematically collect opinions about future dates of 
occurrence of social and technological events. 
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by using a statistical group opinion, 
group pressure toward conformity is 
further reduced, and probably more 
important the opinion of every member ■ 
is reflected in the group response 
■pialkey, 1968b , p.9J . 

Present Delphi procedures suggested by Dalkey (1969) 
involve using the computer, which would enable the time delay 
between completion of a Delphi round and reception of informa- 
tion and opinion feedback to be greatly decreased. However, 
time delay as a variable of environmental complexity, and 
therefore one which may affect human information processing 
levels, is' untested. The effect of variations in time between 
completion of a task and reception of feedback has generally 
been studied in motor skill and problem-solving learning 
situations. This effect has been measured in terms of improve- 
ment of a motor skill or retention of information. There is a 
need for understanding how time delay may effect both individual 
estimations of the time of occurrence of events and also changes 
in these estimations between rounds in a predictive process 
that incorporates iteration. This may lead to a more complete 
understanding of the process of convergence (the phenomenon of 
individual estimations moving closer to the statistical summary 
of group responses on each succeeding round in a Delphi question- 
naire) as displayed by many individuals. (Various explanations 
of convergence have been explored by Dalkey (l968a, 1968b, 1969] 
v.ithout any significant results.) 

Similarly, investigations into zhe interaction of the 
situational factors and dispositional factors within the 
Delphi procedure has been limited. Campbell (1966) and Dalkey 
(1968a, 1968b, 1969) have conducted the most complete evaluation 
of the Delphi technique to date. These experiments, while 
suggesting some possible relationships between personality 
characteristics and individual forecasts, have not fully 
explained why . c ;ome individuals converge (shift their estima- 
tions toward the information feedback) significantly more than 
•others, nor how such shifts affect the final forecast. The 
behavior of the individual (rather than simply the behavior of 
the group as a whole) within the Delphi process, or any predic- 
tive process, must be better understood if realistic interpre- 
tations of the resulting opinions are to be made. Moreover, 
efforts to increase our capacity to think about the future 
must rest on an understanding of ttie process involved, which 
includes a complex interaction of "situational" and "dispositional 
factors. 
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Questions to be Investigated 

A. "WOULD A DELPHI PROCEDURE THAT 
INVOLVES IMMEDIATE (WITHIN 5 
SECONDS) FEEDBACK ELICIT A DIF- 
FERENT SET OF BEHAVIORS THAN A 
TRADITIONAL DELPHI PROCEDURE 
INVOLVING DELAYED FEEDBACK (24 
HOURS LATER)?" 

B. "IN A DELPHI PROCEDURE, WHAT ARE 
THE RELATIONSHIPS THAT EXIST AMONG 
MEASURES OF INTEGRATIVE COMPLEXITY, 

ESTIMATIONS ABOUT THE TIME OF 
OCCURRENCE OF FUTURE EVENTS, AND 
TIME DELAY BETWEEN THE COMPLETION 
OF A TASK AND THE RECEPTION OF 
CONTROLLED FEEDBACK?" 

De f initions 

I* Integrative Complexity : A range of scores from 2 to 14 

derived from the sum of the top cwo rated responses 
(possible rating range, 1 to 7) on the Paragraph Comple- 
tion Test (PCT) (Schroder et al. , 1967). The sum of the 
top two scores will be an individual's integrative 
complexity rating. 

2. High I ntegrative Complexity r A sura of the top two scores 
equal to or greater than 6 on the Paragraph Completion 
Test* 

3. Low Integrative Complexity : A sum of the top two scores 

equal to or lower than 4 on the Paragraph Completion Test* 

4 . Controlled Feedback Round 2 : Controlled feedback of expert 
estimate s randomly adjusted by adding or subtracting 0 to 
10 years. 

5* Controlled Feedback Round 3 : The final median estimate of 

the same group of experts concerning the date of occurrence 
for each event* 

6. Immediate Feedback: Controlled feedback supplied to each 

subject 5 seconds after all subjects have indicated their 
estimations about the occurrence of an event. 

7, Delayed Feedback ; Controlled feedback supplied to each 
subject 24 hours after all subjects have completed a 
Delphi round. 
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8. Estimation Range : The number of years between the 

earliest and latest dates assigned by each subject for 
each event. Each subject will be assigned an estimation 
range score for each of the three rounds of the Delphi 
procedure. This score will be composed of the median of 
the estimation range for all events of that round. 

9. Change in Estimation Range : The difference between each 

subject’s estimation range score for a round and its 
preceding round. 

10. Median Estimation Date : The midpoint of each estimation 

range for each event on all rounds. 

11. Relative Tendency- to- Change Measur e : A measure of an 

individual’s propensity to change his estimations rela- 
tive to controlled feedback II between round one and 
round two, for all events, in relationship to all other 
subjects in the same treatment group. 

The Theoretical Model 



The basic questions in this study were investigated in 
the context of conceptual systems theory as explicated by 
Harvey, Hunt, and Schroder (1961) and Schroder, Driver, and 
Streufert (1967) . Conceptual systems theory is based on the 
belief that "a concept is a system of ordering that serves 
as the mediating linkage between the input side (stimuli) 
and the output side (response) [Harvey et aK , 1961, p.l]. M 
Within this theory, man is perceived as an organism that 
receives external stimuli, structures them, and responds to 
them according to an internalized system of interdependent 
concepts. (Individual conceptual systems are dependent on 
the concepts held and how they are interrelated. Differences 
in perception of concepts and the connotation attributed to 
them result in different information processing structures.) 

And in addition, this internal structure, or conceptual system, 
determines to a great degree what responses can and will occur 
(Harvey e_t al^. , 1961). An individual's information processing 
structure is viewed as having a twofold function: first, it 

selects and screens information from the environment, and 
second, it combines these inputs in specific ways. The first 
function is one of content selection? the second function is 
the '^structural or information-processing variable." 

The manner in which these two functions are carried out 
depends on the individual's concepts and his perception of 
those concepts. For example, some may perceive the concept 
"rules" as connoting rigidity, authority, inflexibility and 
exclusion of alternate meanings. To others, it may connote 
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flexibility, general guidelines and principles relative to 
attending conditions. In some, but not in others, the concept 
"rule" may generate anxiety or hostility. Dependent upon his 
perception, the individual may react by seeking rapid closure 
and avoidance of ambiguity, or, on the other hand, he may 
project far beyond the immediate concept and draw together 
several related concepts. 

The emphasis of conceptual systems theory is on how an 
individual processes information, not on what he thinks. 

This emphasis is helpful in exploring the use of structural 
variables in understanding individual behavior. Schroder 
et al . (1967) stated that 

When the experimental environment is 
sufficiently complex, persons process 
information in different ways under 
different situational conditions, and 
different persons use different ways 
of processing information under the 
same conditions £p.5j. 

They further stated that "Given the same amount of 'information , 
different people use different conceptual rules in thinking, 
deciding, and interrelating Q?.3j." If individuals process 
the same information differently, then, regardless of the 
content of the outcome, different adaptive consequences follow. 

Implications of the Model 

People seem to differ in their degree of conceptual 
complexity, and these differences have been scaled along a 
continuum that goes from levels of concreteness to levels of 
abstractness. The degree of complexity or lack of it determines 
the way in which an individual selects stimuli, integrates them, 
and adapts to them. Schroder et^ al.. (1967) labeled the degree 
of elaborateness of this system "integrative complexity." An 
individual's ability to generate new rules, or new^ ways of 
combining old ones, determines his level of complexity . 

Schroder _e_t al^. (1967) suggested that behavior indicating 
levels of elaborateness in internally structuring stimuli can 
be categorized by the integrative complexity scale. On this 
scale, the "simple" concrete person is assumed to be character- 
ized by: (a) low tolerance for uncertainty and ambiguity, (b) 

limited perspective, (c) limited inclination to project beyond 
the immediacy of present concrete information, and (d) reliance 
on outside rules and guidelines, rather than on an internal 
structure for handling new information. Concrete integrative 
structures are synonymous with fixed rules; i.e., procedures 
for organizing alternate sets of rules are limited. Conversely, 
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the scale has been used to define integrative complexity based 
on certain assumptions of flexibility; i.e., the more complex 
one’s system is, the greater one's ability to generate his own 
rules and guidelines for handling new information, the greater 
one's tolerance for uncertainty and ambiguity, and the greater 
one's potential to generate alternate combinations ‘of informa- 
tion. Complex integrative structures are synonymous with flex- 
ible rules, where many interconnections among rules exist and 
where the capability of generating new rules and combinations 
of rules is an integral part of the structure. The conceptual 
systems model suggests that the more complex one's information 
processing system is, the more varied and elaborate his responses 
will be? and the more concrete one's information processing 
system is, the more constrictive his responses will be. Measures 
of integrative complexity (IC) were used in this study as a 
description of * individual information processing systems. 

Further implications of conceptual systems theory also 
focus on the relationship between environmental complexity 
and conceptual level. As Schroder et aK (1967) stated, “We 
strongly support the view that level of information processing 
is an interactive consequence of dispositional and conditional 
factors Cp.29}.“ 

Simple or concrete environments, which are characterized 
by lack of diversity of input stimuli, do not stimulate 
complex information processing. For example, a task that 
involves estimating which of a series of parallel lines is 
the longest does not require or encourage complex information 
processing. At the other extreme, very complex environments, 
which can be characterized by extremely diverse and numerous 
input stimuli, also fail to stimulate complex information 
processing. For an individual there is an optimum level of 
environmental complexity that stimulates and maximizes his 
information-processing ability. This relationship between 
environmental complexity and level of information processing 
has been defined as the M U curve hypothesis" by Schroder e_t al . 
(1967). 

Rationale for the Model 

This model was used because it supports the assumption 
that the way in which one makes estimations about future events 
is related to the degree of complexity of his use of information, 
and to the level of complexity of the task environment. 

This study proposed that given a task (making estimations 
about the time of occurrence of future events) whose situational 
factors were characterized by: (a) ambiguity and uncertainty, 
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(b) lack of external standards, and (c) lack of concrete 
information, individual performance in this task can be 
explained by a knowledge of the individual's integrative 
complexity rating, as well as a knowledge of the structure 
of the task environment. 

This study also suggests that conceptual level is pre- 
dictive of the performance of individuals who are required 
to think about the future when such a task includes: (a) 

various levels of structural complexity, (b) inputs of 
additional information in the form of a statistical summary 
of a group of experts r responses for an event, and (c) re- 
evaluation of individual estimates in view of the new inputs. 

Procedures 



One hundred and eighteen graduate and upperclass under- 
graduates in the College of Education, Syracuse University, 
were administered the Paragraph Completion Test (PCT) . These 
tests were then rated and the resulting individual integrative 
complexity indices were ranked from high to low and dicnotomized 
into high integrative complexity (HIC) and low integrative 
complexity (LIC) . Those subjects who fell into either category 
were contacted and asked to participate in a further experiment. 
All those subjects who fell into either of the two categories 
and who agreed to further participate were randomly assigned to 
treatment one (Ti) or treatment two (T2) • 

Each treatment group was given a date on wl ich to meet 
with the experimenter. T^ met for one session in which they 
proceeded through the three Delphi rounds. Each subject in T^ 
was requested to make earliest and latest estimations for an 
event in round one (R^) . Five seconds later, he received 
■ controlled feedback round II for that event and was asked to re- 
evaluate his estimations for che same event in round two (R2) . 
Controlled feedback round III was then given to the subject 
with a request to re-evaluate his R2 estimations in round 
three (R3) . 

T2 met on three consecutive afternoons for fifteen minutes. 
Rl was completed on the first afternoon, Rp was completed on 
the second afternoon, and R3 was completed ’on the third 
afternoon. 

All subjects were required to complete a Delphi question- 
naire that was composed of eight future events taken from the 
Gordon and Helmer (1966) Delphi experiment at Rand Corporation. 
Examples of these events are 
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Chemical control of the aging process, 
permitting extension of life span by 
fifty years. 

Man-machine symbiosis, enabling man to 
extend his intelligence by direct 
electromechanical interaction between 
his brain and a computing machine. 

The data collected met the basic criteria for non- 
parcimetric treatment (Siegal, 1956). The data were ordinal 
(i.e., the data could be ranked because scores represent 
strength of ranks drawn from a continuous distribution) . The 
following non-parametric tests were used in this study: the 

Sign Test, the Median Test^ and the Mann-Whitney U Test. An 
OC = .Of level of confidence was used to decide whether differ- 
ences between samples are significant. 



Hypotheses and Findings 

Hi: LICs WILL PRODUCE HIGHER SCORES 

ON THE TWO DEPENDENT VARIABLES — 

ESTIMATION RANGE, AND RELATIVE 
TENDENCY-TO-CHANGE — THAN WILL 
HICs. 

The expectation that the differential effects of rounds, 
regardless of time delay of information feedback, would cause 
low integrative ly complex persons (LICs) to produce signifi- 
cantly higher scores on the two dependent variables than high 
integratively complex persons (HICs) was completely supported 
by the data. 

H2J A DELPHI PROCEDURE THAT INVOLVES 
IMMEDIATE INFORMATION FEEDBACK 
(5 SECONDS) WILL ELICIT A DIFFERENT 
SET OF ESTIMATION RANGE SCORES ON 
ROUND TWO AND ON ROUND THREE, AND 
RELATIVE TENDENCY-TO-CHANGE SCORES 
FROM HICs, THAN A DELPHI PROCEDURE 
THAT INVOLVES DELAYED INFORMATION 
X FEEDBACK (24 HOURS). 

» 

The expectation that the differential effects of time 
delay of feedback on high integratively complex persons would 
cause significant differences in performance between treatment 
one (immediate information feedback) HICs and treatment two 
(delayed information feedback) HICs was supported on the basis 
of the first dependent variable. But this expectation was 
not supported on the basis of the third dependent variable 
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data. Ti HICs differed significantly from T 2 HICs on the 
basis of estimation range scores on rounds two and three. 

With delayed feedback T2 HICs produced higher scores than 
Ti HICs. T] HICs did not differ significantly from T 2 HICs 
in their propensity -to' move toward the controlled feedback 
as indicated by their relative tendency-to-chanqe scores. 
Differential time delay of information feedback did not 
significantly affect high integrat ively complex subjects' 
propensity to change their estimations in the direction of 
the controlled feedback. One may conclude that an individ- 
ual's propensity to change in the direction of information 
feedback is a personality characteristic that is relatively 
uneffected by differences in experimental structure. 

H 2 a : TREATMENT ONE HICs WILL NOT DIFFER 

SIGNIFICANTLY FROM TREATMENT TWO 
HICs ON ROUND ONE ESTIMATION RANGE 
SCORES. 

The expectation that procedural differences between 
treatment one and treatment two*, other than differential 
time delay, would not significantly effect high integratively 
complex subjects' performance was supported. HICs did not 

differ significantly from T2 HICs on round one estimation 
range scores. 

H3: A DELPHI PROCEDURE THAT INVOLVES 

IMMEDIATE INFORMATION FEEDBACK 
(5 SECONDS) WILL NOT ELICIT A 
DIFFERENT SET OF ESTIMATION RANGE 
SCORES ON ROUND TWO AND ON ROUND 
THREE, AND RELATIVE TENDENCY-TO- 
CHANGE SCORES FROM LICs , THAN A 
DELPHI PROCEDURE THAT INVOLVES 
. DELAYED INFORMATION FEEDBACK 
. ’ , " (24 HOURS) . 



^Subjects in Ti were required to make earliest and 
latest estimates for an event on Ru they then received 
controlled feedback and were required to go through R2 . 

Tj^ subjects then went through R3. At the completion of the 
three Delphi rounds for an event, Tg subjects then went on 
to Ri on the next event. This procedure was repeated until 
all eight events had been considered. 

Subjects in T2 went through R^ for all eight events 
sequentially. Twenty-four hours later, they went through R2 
for all eight events. Twenty-four hours later, this procedure 
was repeated for R3. 
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The expectation that differential time delay of informa- 
tion feedback would not significantly effect low integratively 
complex persons was supported. Ti LICs did not differ from T 2 
LICs on the first and the second dependent variables. The 
estimation range scores on rounds two and three were similar 
for both Ti and T? LICs. The propensity of LICs to change in 
the direction of the controlled feedback, as indicated by the 
relative tendency-to-change scores, was not significantly 
effected by the differences in time delay of information feed- 
back. One may conclude that differences in experimental 
structure did not significantly effect LICs. 

H 3 a s Ti LICs WILL NOT DIFFER FROM T2 LICs 
ON ROUND ONE ESTIMATION RANGE SCORES. 

The expectation of no significant differences due to 
procedural differences other than differential time delay was' 
completely supported by the* data.. 

‘ H4 : LICs WILL PRODUCE DIFFERENT RELATIVE 

TENDENCY-TO-CHANGE SCORES (Ci) THAN 
HICs JN TREATMENT ONE. 

H5: LICs WILL PRODUCE DIFFERENT RELATIVE 

TENDENCY-TO- CHANGE SCORES (C^) TH A N 
HICs IN TREATMENT TWO. 

It was expected that the findings of testing on the 
br.sis of relative tendency- to-change scores would hold true 
within treatment one and within treatment two. The propensity 
of low integratively complex subjects to change their estima- 
tions in the direction of the controlled feedback, as reflected 
in their relative tendency-to-chunge scores, was significantly 
greater than high integratively complex subjects 1 propensity 
to change their estimations in the direction of the controlled 
feedback, as reflected in their relative tendency-to-change 
scores in each of the treatments separately. 



Conclusions 



Three major conclusions may be drawn from the results 
of this study: 

(a) In the Delphi process, integrative complexity is 
strongly predictive of performance on the tasks of making 
earliest and latest estimates about the time of occurrence 
of remote future events, and changing those estimates in 
light of information feedback. 
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(1) In a belphi process involving remote future 
events, low integratively complex persons 
produce larger estimation ranges (the number 
of years between their earliest and latest 
estimation dates is greater) than do high 
integratively complex persons. 

(2) Low integratively complex persons produce 
larger changes in estimation ranges than do 
high integratively complex persons in a Delphi 
process involving remote future events. 

(b) In the Delphi process, integrative complexity is 
consistently predictive oi convergence. 

(1) Regardless of differential time delay effects, 
low integratively complex persons display a 
greater propensity to change their estimations 
towards external information feedback than do 
high integratively complex persons. 

(c) Differential time delay of information feedback does 
have a significant effect on the performance of individuals 
involved in the Delphi process* 

(1) Differential time delay of information feed- 
back significantly effects the performance 
of high integratively complex persons in. the 
Delphi process on the tasks of making earliest 
and latest estimates and on the task of chang- 
ing those estimates* 

(2) In the Delphi process, differential time 
delay of information feedback significantly 
effects the performance of low integratively 
complex persons on the task of changing the: r 
earliest and latest estimates* 



Discussions of Conceptual Systems Theor y as it Relates t o 
Differential Time Delay in the Delphi Process 

The results of this study clearly support the predictive 
value of conceptual systems theory The ideal characteriza- 
tion of both the high integratively complex and the low 
integratively complex person by the conceptual systems model 
provided a sound rationale for statement of the hypotheses. 
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The major contribution of the model lies in its recognition 
of the interrelatedness of both the dispositional and 
situational factors as zhey bear on observable behavior. 

In this study, this relationship was clearly demonstrated. 

The situational factor of differential time delay of 
feedback significantly effected both high integratively com- 
plex and low integratively complex persons' performances in 
the Delphi process. When immediate information feedback was 
employed, the difference between the performance of LICs and 
HICs was maximized. When delayed information feedback was 
used, the difference between LICs and HICs in performance was 
minimized. The experimental treatments (differential time 
delay) can be assumed to fall somewhere on a continuum from 
low to high environmental complexity (see Figure 8). Treat- 
ment one (Ti) (immediate information feedback) can he assumed 
to represent the optimal experimental environment for differ- 
entiating the performance of HICs and LICs. Treatment two 
(T^ ) (delayed information feedback) can be assumed to represent 
the more simplified of the two experimental environments, 
because of its employment of delayed information feedback 
(24 hours between rounds) . Treatment one involved immediate 
information feedback, whi< h inn y bn assumed to increase 
environmental complexity by placing added pressure on the 
subjects to fun*, t ion in shoi t periods of time (5 seconds 
between rounds). One may conclude that HICs process complex 
in format i on at a fastoL rate arid mu to consistently than LICs. 



Integrative 

Complexity 



Figure 8. 
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Where HICs represents the theoretical relationship berween 
high integratively complex persons and environ- 
mental complexity 

LICs represents the theoretical relationship between 
low integratively complex persons and environ- 
mental complexity.. 

As Schroder et cQ. (1967) indicated, the more simplified 
the environment, the more simplified the information process- 
ing becomes. In other words, if the assumption that T 2 
represents the more simplified of the two experimental' 
environments is valid, and if it is overly simple in such a 
way that simple information processing structures are 
sufficient to cope with it, then one could expect that the 
differences in performance between HICs and LICs would be 
minimized. The data supports this explanation; no signifi- 
cant differences in performance were found between HICs and 
LICs in treatment two. If the assumption is valid that Ti 
represents the more complex of the two experimental environ- 
ments, and if it is complex enough to stimulate complex 
information processing structures, then one could expect that 
the differences in performance between HICs and LICs would be 
maximized. Again, the data supported this explanation. HICs 
differed significantly from LICs in treatment one. 

Evaluating the Outcomes of the Delphi Process 

The findings and conclusions of this study bring to bear 
information that is useful in interpreting the results of the 
Delphi process. They also help to explain the phenomenon of 
convergence, and thereby challenge some of the general 
assumptions underlying the Delphi process. 

The major assumption of the Delphi procedure i6 that 
throu rh its use of (a) anonymity, (b) controlled feedback, 
and (c) statistical group responses, it somehow reduces the 
effect of 1 non-objective” variables found in direct confron- 
tation group processes, Some of these non-objective variables 
are dominant personality influence, group pressure to conform, 
resistance to change a publicly stated position, and predis- 
position to depreciate stated opinions of others based on 
personality factors. Dalkey (1969), Dalkey and Helmer (1963), 
Campbell (1966) , and many others tended to view the results 
of a Delphi exercise as being more acceptable than the 
consensus arrived at by interpersonal group processes. More 
specifically, the phenomenon of convergence in the direction 
of some "correct solution in the Delphi process is regarded 
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as being a product of ‘objectivity relatively free from inter- 
ference of certain psychological variables. In view of the 
conclusions and findings of this study, that assumption is 
unfounded . 

This study, within the stated limitations, concluded 
that performance in the Delphi process is predictable on the 
basis of knowledge of individuals* integrative complexity 
ratings. Conceptual systems theory indicates that low 
integratively complex persons may be characterized by: (a) 

low tolerance for uncertainty and ambiguity, (b) limited 
perspective, <c) limited inclination to project beyond the 
immediacy of present concrete information, and (d) reliance 
on external information and guidelines. From this descrip- 
tion, one would assume that, the low integratively complex 
person would, in the Delphi process, rely heavily on infor- 
mation feedback and be significantly influenced by it. The 
results of this study clearly supported the above expectation. 
Low integratively complex persons converged in the direction 
of the controlled information feedback significantly more than 
high integratively complex persons. In addition, the perform- 
ance of individuals in the Delphi process on the tasks of (a) 
making earliest and latest estimations, and (b) changing those 
estimations in light of the controlled feedback, was found to 
be predictable on the basis of integrative complexity. Again, 
low integratively complex subjects produced significantly 
broader estimation ranges and significantly large*- changes in 
estimations. 

In summary, the implications of these findings are 
relevant to evaluating outcomes of the Delphi process. High 
integratively complex persons are more likely to base their 
estimates on something other than non-objective influences. 

Low integratively complex individuals consistently tend to 
change their estimates in the direction of the controlled 
feedback, and are more likely to be influenced by external 
inputs. In general, the outcomes of a Delphi exercise are as 
reflective of those non-objective factors as the outcomes of 
direct confrontation groups. As long as a process involves 
feedback, one may assume that some individuals will shift in 
the direction of that external input based on personality 
structure rather than on the basis of rational, objective 
evaluation. 

Differential Time Delay and Delphi 1 

In addition, the conclusion that differential time delay 
Of information feedback has a significant effect on the 
performance of individuals in a Delphi process brings to 
light another area of potential concern. One of the major 
difficulties with the traditional Delphi procedure has been 
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with the rather long delay between completion of a Delphi 
round, the reception of the controlled feedback, and the 
beginning of the following round. This time delay is 
usually caused by the need to receive the questionnaires, 
summarize the responses, and then return the questionnaires 
to the participants for the next round. The application of 
computer techniques to the Delphi process presents a very 
effective management technique for reducing the manipulative 
problems of the traditional process. The computerization of 
the Delphi process has been strongly urged by practitioners 
and is presently being implemented. 

Before computerization of the Delphi process is imple- 
mented, extensive investigation of a large variety of 
different time delays should be carried out. In view of 
the fact that various time delays do have an effect, a more 
complete understanding of the parameters of this effect is 
required in order to more fully understand the related differ-' 
ences in individual performances. Without this understanding, 
evaluation of the outcomes of the process may be impossible. 

In the present study, the use of immediate information 
feedback maximized the differences between high integratively 
complex and low integratively complex subjects. Conversely, 
when delayed information feedback was used, high integratively 
complex and low integratively complex subjects performed 
alike. It is impossible, from the findings of the present 
study, to assume that no other time delay increments than 
those used in this study will have significant effect. One 
can assume that many different combinations will have a 
variety of different effects on individuals r performances 
within the Delphi process. Failure to recognize the inter- 
relatedness of time delay variables and performance in the 
Delphi process may culminate in a misinterpretation of the 
outcomes, whether they be a product of a computerized process 
or a more traditional questionnaire procedure. 

The results of this study further imply that the Delphi 
process may have greater value when it is used as a pedagogical 
tool for providing a framework in which Individuals are 
motivated to think about the future and to full together as 
objectively as possible a large variety of information. This 
study clearly supports che value of the Delphi process as an 
analytical tool for the study of human behavior in highly^ 
complex and ambiguous situations. ' 

It is fundamental that in order to improve forecasts, we 
have to improve our capacity to think about the future. To do 
this, we must know more about the process of thinking about 
the future. In this regard the Delphi process does two things. 
It requires people to think about the future, and it requires 
them to do it in complex ways. 
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V. 
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The greatest danger lies in ascribing to the outcomes of 
the Delphi process some higher level of plausibil ity , that is,, 
equating convergence with plausibility, than is warranted in 
view of the findings of not only the present study, but also 
the studies of Weaver (1969) and Campbell (1966). 




37 



BIBLIOGRAPHY 



Campbell , 

Dalkey, N 
Dalkey, N 
Dalkey, N 

Harvey, 0 

Halmer, 0 
Helmer, 0 

Schroder , 
Waldron , 

^Weaver, W 



'A 



R.M., A methodological study of the utilization of 
experts. {Doctoral dissertation, University 
of California, Los Angeles.) An Arbor, Mich.: 
University Microfilms, 1966, No. 67-4491. 

.C., Experiments in group prediction. Rand Corp., 

1968, P-3820 (a) . 

.C., Predicting the future. Rand Corp., 1968, 

P-3948 (b) . 

.C., The Delphi method: An experimental study of 

group opinion. Rand Corp., June 1169. Memo- 
randum RM-5888-PR. 

.J. , Hunt, D.E. , and Schroder, H.M., Co nceptual 

systems and personality "organisation . N ev; York: 
John Wiley and Sons, ~T961. 

. J. , Social technology . New York: Basic Books, 1966. 

.J., Analysis of the future: The Delphi method. 

Rand Corp., March 1967, P-3558. 

H. , Driver, M. and Streufert, S. , Human information 

processing. New York: Holt, Rinehart and Winston 

1967. 

J.S., An investigation into the relationships among 
conceptual level, time delay of information 
feedback, and performance in the Delphi process. 
(Doctoral dissertation, Syracuse University, 

New York.) Ann Arbor, Mich.: University 

Microfilms , 1970 . 

. T . , An exploration into the re lationship' between 

conceptual level and forecasting future events. 
(Doctoral dissertation, Syracuse University, 

New York.) Ann Arbor, Mich.: University 

' s Microfilms, 1969. 



